1 link tagged with all of: observability + resource optimization + salesforce data 360 + dataflint
Click any tag below to further narrow down your results
Links
Salesforce's Data 360 team processed eight million daily Spark jobs and discovered that successful execution doesn't mean efficient resource use. They built instrumentation to measure actual CPU, memory, and executor usage without changing application code, then used those metrics to cut compute costs 31% while improving job throughput.
- Successful job completion masks resource waste: the team found workloads requesting 40 executors where most did zero useful work, yet the job still succeeded, making waste invisible without detailed telemetry.
- They injected a DataFlint Spark plugin through a shared submission gateway (DPC) to collect metrics via Kafka into an Iceberg lakehouse, avoiding the need to instrument hundreds of application teams' code.
- Raw metrics mislead without careful interpretation: low executor utilization can signal either oversized allocation or a shuffle bottleneck requiring different fixes; they built a curated analytical layer (Silver) to distinguish waste from legitimate bottlenecks.
- Reducing initialExecutors from 40 to 4 with dynamic allocation and lowering max-executor limits by 20% cut weekly compute costs 31% over a month in production with no runtime, retry, or failure regressions.